I have executed the following queries in order to have an incremental append in my hive table "customers" from the mysql table "CUSTOMERS".
--- Sqoop command to create a saved job for the required import ----
sqoop job --create myIncreImportjob -- import --connect jdbc:mysql://localhost/training --username training --password training \
--table CUSTOMERS --hive-import --hive-table customers --incremental append --check-column ID --last-value 4 --hive-table customers --fields-terminated-by ',' ;
--- Command for executing the saved job
sqoop job --exec myIncreImportjob;
Observations:
1. If you don't mention the --hive-import, the data will be imported to directory with the mysql table name.
2. If we don't mention the --last-value, entire table data will be imported.
3. Even if the saved JOB has the hard-coded value 4 as last value, next time the value is overridden by the last value (from ID column) thrown after sqoop execution.
--- Sqoop command to create a saved job for the required import ----
sqoop job --create myIncreImportjob -- import --connect jdbc:mysql://localhost/training --username training --password training \
--table CUSTOMERS --hive-import --hive-table customers --incremental append --check-column ID --last-value 4 --hive-table customers --fields-terminated-by ',' ;
--- Command for executing the saved job
sqoop job --exec myIncreImportjob;
Observations:
1. If you don't mention the --hive-import, the data will be imported to directory with the mysql table name.
2. If we don't mention the --last-value, entire table data will be imported.
3. Even if the saved JOB has the hard-coded value 4 as last value, next time the value is overridden by the last value (from ID column) thrown after sqoop execution.
