進入 sqoop2-shell
[hadoop@netkiller ~]$ sqoop2-shell Setting conf dir: /srv/apache-sqoop/bin/../conf Sqoop home directory: /srv/apache-sqoop Sqoop Shell: Type 'help' or '\h' for help. sqoop:000>
Sqoop client script:
sqoop2-shell /path/to/your/script.sqoop
sqoop:000> show version client version: Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
sqoop:000> show version --all client version: Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016 0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable server version: Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016 API versions: [v1]
sqoop:000> set server --host master --port 12000 --webapp sqoop Server is set successfully
sqoop:000> show connector 0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable +------------------------+---------+------------------------------------------------------------+----------------------+ | Name | Version | Class | Supported Directions | +------------------------+---------+------------------------------------------------------------+----------------------+ | generic-jdbc-connector | 1.99.7 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO | | kite-connector | 1.99.7 | org.apache.sqoop.connector.kite.KiteConnector | FROM/TO | | oracle-jdbc-connector | 1.99.7 | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO | | ftp-connector | 1.99.7 | org.apache.sqoop.connector.ftp.FtpConnector | TO | | hdfs-connector | 1.99.7 | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO | | kafka-connector | 1.99.7 | org.apache.sqoop.connector.kafka.KafkaConnector | TO | | sftp-connector | 1.99.7 | org.apache.sqoop.connector.sftp.SftpConnector | TO | +------------------------+---------+------------------------------------------------------------+----------------------+ sqoop:000> sqoop list-databases --connect jdbc:mysql://192.168.1.1:3306/ --username root --password 123456
sqoop:000> show connector --all
7 connector(s) to show:
Connector with Name: generic-jdbc-connector
Class: 1.99.7
Version: FROM/TO
Supported Directions {4}
link config 1:
Name: linkConfig
Label: Database connection
Help: Contains configuration that is required to establish connection with your database server.
Input 1:
Name: linkConfig.jdbcDriver
Label: Driver class
Help: Fully qualified class name of the JDBC driver that will be used for establishing this connection. Check documentation for instructions how to make the driver's jar files available to Sqoop 2 server.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 128
Input 2:
Name: linkConfig.connectionString
Label: Connection String
Help: JDBC connection string associated with your database server.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 3:
Name: linkConfig.username
Label: Username
Help: Username to be used for connection to the database server.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 40
Input 4:
Name: linkConfig.password
Label: Password
Help: Password to be used for connection to the database server.
Type: STRING
Sensitive: true
Editable By: ANY
Overrides:
Size: 40
Input 5:
Name: linkConfig.fetchSize
Label: Fetch Size
Help: Optional hint specifying requested JDBC fetch size.
Type: INTEGER
Sensitive: false
Editable By: ANY
Overrides:
Input 6:
Name: linkConfig.jdbcProperties
Label: Connection Properties
Help: Key-value pairs that should be passed down to JDBC driver when establishing connection.
Type: MAP
Sensitive: false
Editable By: ANY
Overrides:
link config 2:
Name: dialect
Label: SQL Dialect
Help: Database dialect that should be used for generated queries.
Input 1:
Name: dialect.identifierEnclose
Label: Identifier enclose
Help: Character(s) that should be used to enclose table name, schema or column names.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 5
FROM Job config 1:
Name: fromJobConfig
Label: Database source
Help: Specifies source and way how the data should be fetched from source database.
Input 1:
Name: fromJobConfig.schemaName
Label: Schema name
Help: Schema name if the table is not stored in default schema. Note: Not all database systems understands the concept of schema.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 50
Input 2:
Name: fromJobConfig.tableName
Label: Table name
Help: Input table name from from which data will be retrieved.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 50
Input 3:
Name: fromJobConfig.sql
Label: SQL statement
Help: Import data from given query's results set rather then static table.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 4:
Name: fromJobConfig.columnList
Label: Column names
Help: Subset of columns that should be retrieved from source table.
Type: LIST
Sensitive: false
Editable By: ANY
Overrides:
Input 5:
Name: fromJobConfig.partitionColumn
Label: Partition column
Help: Input column that should be use to split the import into independent parallel processes. This column will be used in condition of generated queries.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 50
Input 6:
Name: fromJobConfig.allowNullValueInPartitionColumn
Label: Partition column nullable
Help: Set true if partition column can contain NULL value.
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 7:
Name: fromJobConfig.boundaryQuery
Label: Boundary query
Help: Customize query to retrieve minimal and maximal value of partition column.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 50
FROM Job config 2:
Name: incrementalRead
Label: Incremental read
Help: Configures optional incremental read from the database where source data are changing over time and only new changes need to be re-imported.
Input 1:
Name: incrementalRead.checkColumn
Label: Check column
Help: Column that is checked during incremental read for new values.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 50
Input 2:
Name: incrementalRead.lastValue
Label: Last value
Help: Last imported value, job will read only newer values.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: -1
TO Job config 1:
Name: toJobConfig
Label: Database target
Help: Describes target destination and way how data should be persisted on the RDBMS system.
Input 1:
Name: toJobConfig.schemaName
Label: Schema name
Help: Schema name if the table is not stored in default schema. Note: Not all database systems understands the concept of schema.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 50
Input 2:
Name: toJobConfig.tableName
Label: Table name
Help: Destination table name to store transfer results.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 3:
Name: toJobConfig.columnList
Label: Column names
Help: Subset of columns that will will be written to. Omitted columns have to either allow NULL values or have defined default value.
Type: LIST
Sensitive: false
Editable By: ANY
Overrides:
Input 4:
Name: toJobConfig.stageTableName
Label: Staging table
Help: Name of table with same structure as final table that should be used as a staging destination. Data will be directly written to final table if no staging table is specified.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 5:
Name: toJobConfig.shouldClearStageTable
Label: Clear stage table
Help: If set to true, staging table will be wiped out upon job start.
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Connector with Name: kite-connector
Class: 1.99.7
Version: FROM/TO
Supported Directions {4}
link config 1:
Name: linkConfig
Label: Global configuration
Help: Global configuration options that will be used for both from and to sides.
Input 1:
Name: linkConfig.authority
Label: HDFS host and port
Help: Optional to override HDFS file system location.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 2:
Name: linkConfig.confDir
Label: Hadoop conf directory
Help: Directory with Hadoop configuration files. This directory will be added to the classpath.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
FROM Job config 1:
Name: fromJobConfig
Label: Source configuration
Help: Configuration options relevant to source dataset.
Input 1:
Name: fromJobConfig.uri
Label: dataset:hdfs://namespace/table
Help: Kite Dataset URI from which data will be read.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
TO Job config 1:
Name: toJobConfig
Label: Target configuration
Help: Configuration options relevant to target dataset.
Input 1:
Name: toJobConfig.uri
Label: Dataset URI
Help: Kite Dataset URI where should be data written to.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 2:
Name: toJobConfig.fileFormat
Label: File format
Help: Storage format that should be used when creating new dataset.
Type: ENUM
Sensitive: false
Editable By: ANY
Overrides:
Possible values: CSV,AVRO,PARQUET
Connector with Name: oracle-jdbc-connector
Class: 1.99.7
Version: FROM/TO
Supported Directions {4}
link config 1:
Name: connectionConfig
Label: Oracle connection configuration
Help: You must supply the information requested in order to create an Oracle connection object.
Input 1:
Name: connectionConfig.connectionString
Label: JDBC connection string
Help: Enter the value of JDBC connection string to be used by this connector for creating Oracle connections.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 128
Input 2:
Name: connectionConfig.username
Label: Username
Help: Enter the username to be used for connecting to the database.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 40
Input 3:
Name: connectionConfig.password
Label: Password
Help: Enter the password to be used for connecting to the database.
Type: STRING
Sensitive: true
Editable By: ANY
Overrides:
Size: 40
Input 4:
Name: connectionConfig.jdbcProperties
Label: JDBC connection properties
Help: Enter any JDBC properties that should be supplied during the creation of connection.
Type: MAP
Sensitive: false
Editable By: ANY
Overrides:
Input 5:
Name: connectionConfig.timeZone
Label: Session time zone
Help: timeZone
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: -1
Input 6:
Name: connectionConfig.actionName
Label: Session action name
Help: actionName
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: -1
Input 7:
Name: connectionConfig.fetchSize
Label: JDBC fetch size
Help: fetchSize
Type: INTEGER
Sensitive: false
Editable By: ANY
Overrides:
Input 8:
Name: connectionConfig.initializationStatements
Label: Session initialization statements
Help: initializationStatements
Type: LIST
Sensitive: false
Editable By: ANY
Overrides:
Input 9:
Name: connectionConfig.jdbcUrlVerbatim
Label: Use JDBC connection string verbatim
Help: jdbcUrlVerbatim
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 10:
Name: connectionConfig.racServiceName
Label: RAC service name
Help: racServiceName
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: -1
FROM Job config 1:
Name: fromJobConfig
Label: From Oracle configuration
Help: You must supply the information requested in order to create the FROM part of the job object.
Input 1:
Name: fromJobConfig.tableName
Label: Table name
Help: tableName
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 2:
Name: fromJobConfig.columns
Label: Columns
Help: Columns
Type: LIST
Sensitive: false
Editable By: ANY
Overrides:
Input 3:
Name: fromJobConfig.consistentRead
Label: Consistent read
Help: consistentRead
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 4:
Name: fromJobConfig.consistentReadScn
Label: Consistent read SCN
Help: consistentReadScn
Type: LONG
Sensitive: false
Editable By: ANY
Overrides:
Input 5:
Name: fromJobConfig.partitionList
Label: Partitions
Help: partitionList
Type: LIST
Sensitive: false
Editable By: ANY
Overrides:
Input 6:
Name: fromJobConfig.dataChunkMethod
Label: Data chunk method
Help: dataChunkMethod
Type: ENUM
Sensitive: false
Editable By: ANY
Overrides:
Possible values: ROWID,PARTITION
Input 7:
Name: fromJobConfig.dataChunkAllocationMethod
Label: Data chunk allocation method
Help: dataChunkAllocationMethod
Type: ENUM
Sensitive: false
Editable By: ANY
Overrides:
Possible values: ROUNDROBIN,SEQUENTIAL,RANDOM
Input 8:
Name: fromJobConfig.whereClauseLocation
Label: Where clause location
Help: whereClauseLocation
Type: ENUM
Sensitive: false
Editable By: ANY
Overrides:
Possible values: SUBSPLIT,SPLIT
Input 9:
Name: fromJobConfig.omitLobColumns
Label: Omit LOB columns
Help: omitLobColumns
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 10:
Name: fromJobConfig.queryHint
Label: Query hint
Help: queryHint
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 11:
Name: fromJobConfig.conditions
Label: Conditions
Help: conditions
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
TO Job config 1:
Name: toJobConfig
Label: To database configuration
Help: You must supply the information requested in order to create the TO part of the job object.
Input 1:
Name: toJobConfig.tableName
Label: Table name
Help: Table name to write data into
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 2:
Name: toJobConfig.columns
Label: Columns
Help: Columns
Type: LIST
Sensitive: false
Editable By: ANY
Overrides:
Input 3:
Name: toJobConfig.templateTable
Label: Template table name
Help: templateTable
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 4:
Name: toJobConfig.partitioned
Label: Partitioned
Help: partitioned
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 5:
Name: toJobConfig.nologging
Label: Nologging
Help: nologging
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 6:
Name: toJobConfig.updateKey
Label: Update key columns
Help: updateKey
Type: LIST
Sensitive: false
Editable By: ANY
Overrides:
Input 7:
Name: toJobConfig.updateMerge
Label: Merge updates
Help: updateMerge
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 8:
Name: toJobConfig.dropTableIfExists
Label: Drop table if exists
Help: dropTableIfExists
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 9:
Name: toJobConfig.storageClause
Label: Template table storage clause
Help: storageClause
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 10:
Name: toJobConfig.temporaryStorageClause
Label: Temporary table storage clause
Help: temporaryStorageClause
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 2000
Input 11:
Name: toJobConfig.appendValuesHint
Label: Append values hint usage
Help: appendValuesHint
Type: ENUM
Sensitive: false
Editable By: ANY
Overrides:
Possible values: AUTO,ON,OFF
Input 12:
Name: toJobConfig.parallel
Label: Parallel
Help: parallel
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Connector with Name: ftp-connector
Class: 1.99.7
Version: TO
Supported Directions {4}
link config 1:
Name: linkConfig
Label: FTP Server configuration
Help: Parameters required to connect to an FTP server.
Input 1:
Name: linkConfig.server
Label: Hostname
Help: Hostname for the FTP server.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 256
Input 2:
Name: linkConfig.port
Label: Port
Help: Port for the FTP server. Connector will use 21 if omitted.
Type: INTEGER
Sensitive: false
Editable By: ANY
Overrides:
Input 3:
Name: linkConfig.username
Label: Username
Help: Username that will be used to authenticate connection to the FTP Server.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 256
Input 4:
Name: linkConfig.password
Label: Password
Help: Password that will be used to authenticate connection to the FTP Server.
Type: STRING
Sensitive: true
Editable By: ANY
Overrides:
Size: 256
TO Job config 1:
Name: toJobConfig
Label: Output configuration
Help: Parameters required to store data on the FTP server.
Input 1:
Name: toJobConfig.outputDirectory
Label: Output directory
Help: Directory on the FTP server to write data to.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 260
Connector with Name: hdfs-connector
Class: 1.99.7
Version: FROM/TO
Supported Directions {4}
link config 1:
Name: linkConfig
Label: HDFS cluster
Help: Contains configuration required to connect to your HDFS cluster.
Input 1:
Name: linkConfig.uri
Label: URI
Help: Namenode URI for your cluster.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 2:
Name: linkConfig.confDir
Label: Conf directory
Help: Directory on Sqoop server machine with hdfs configuration files (hdfs-site.xml, ...). This connector will load all files ending with -site.xml.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 3:
Name: linkConfig.configOverrides
Label: Additional configs:
Help: Additional configuration that will be set on HDFS Configuration object, possibly overriding any keys loaded from configuration files.
Type: MAP
Sensitive: false
Editable By: ANY
Overrides:
FROM Job config 1:
Name: fromJobConfig
Label: Input configuration
Help: Specifies information required to get data from HDFS.
Input 1:
Name: fromJobConfig.inputDirectory
Label: Input directory
Help: Input directory containing files that should be transferred.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 2:
Name: fromJobConfig.overrideNullValue
Label: Override null value
Help: If set to true, then the null value will be overridden with the value set in Null value.
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 3:
Name: fromJobConfig.nullValue
Label: Null value
Help: For file formats that doesn't have native representation of NULL (as for example text file) use this particular string to decode NULL value.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
FROM Job config 2:
Name: incremental
Label: Incremental import
Help: Information relevant for incremental reading from HDFS.
Input 1:
Name: incremental.incrementalType
Label: Incremental type
Help: Type of incremental import.
Type: ENUM
Sensitive: false
Editable By: ANY
Overrides:
Possible values: NONE,NEW_FILES
Input 2:
Name: incremental.lastImportedDate
Label: Last imported date
Help: Datetime stamp of last read file. Next job execution will read only files that have been created after this point in time.
Type: DATETIME
Sensitive: false
Editable By: ANY
Overrides:
TO Job config 1:
Name: toJobConfig
Label: Target configuration
Help: Configuration describing where and how the resulting data should be stored.
Input 1:
Name: toJobConfig.overrideNullValue
Label: Override null value
Help: If set to true, then the null value will be overridden with the value set in Null value.
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Input 2:
Name: toJobConfig.nullValue
Label: Null value
Help: For file formats that doesn't have native representation of NULL (as for example text file) use this particular string to encode NULL value.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 3:
Name: toJobConfig.outputFormat
Label: File format
Help: File format that should be used for transferred data.
Type: ENUM
Sensitive: false
Editable By: ANY
Overrides:
Possible values: TEXT_FILE,SEQUENCE_FILE,PARQUET_FILE
Input 4:
Name: toJobConfig.compression
Label: Compression codec
Help: Compression codec that should be use to compress transferred data.
Type: ENUM
Sensitive: false
Editable By: ANY
Overrides:
Possible values: NONE,DEFAULT,DEFLATE,GZIP,BZIP2,LZO,LZ4,SNAPPY,CUSTOM
Input 5:
Name: toJobConfig.customCompression
Label: Custom codec
Help: Fully qualified class name with Hadoop codec implementation that should be used if none of the build-in options are suitable.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 6:
Name: toJobConfig.outputDirectory
Label: Output directory
Help: HDFS directory where transferred data will be written to.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 7:
Name: toJobConfig.appendMode
Label: Append mode
Help: If set to false, job will fail if output directory already exists. If set to true then imported data will be stored to already existing and possibly non empty directory.
Type: BOOLEAN
Sensitive: false
Editable By: ANY
Overrides:
Connector with Name: kafka-connector
Class: 1.99.7
Version: TO
Supported Directions {4}
link config 1:
Name: linkConfig
Label: Kafka cluster
Help: Configuration options describing Kafka cluster.
Input 1:
Name: linkConfig.brokerList
Label: Kafka brokers
Help: Comma-separated list of Kafka brokers in the form of host:port. It doesn't need to contain all brokers, but at least two are recommended for high availability
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 1024
Input 2:
Name: linkConfig.zookeeperConnect
Label: Zookeeper quorum
Help: Address of Zookeeper used by the Kafka cluster. Usually host:port. Multiple zookeeper nodes are supported. If Kafka is stored in its own znode use host:portkafka
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
TO Job config 1:
Name: toJobConfig
Label: Output configuration
Help: Configuration necessary when writing data to Kafka.
Input 1:
Name: toJobConfig.topic
Label: Topic
Help: Name of Kafka topic where data will be written into.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Connector with Name: sftp-connector
Class: 1.99.7
Version: TO
Supported Directions {4}
link config 1:
Name: linkConfig
Label: FTP Server configuration
Help: Parameters required to connect to an SFTP server.
Input 1:
Name: linkConfig.server
Label: Hostname
Help: Hostname of the SFTP server.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 255
Input 2:
Name: linkConfig.port
Label: Port
Help: Port for the SFTP server. Connector will use 22 if omitted.
Type: INTEGER
Sensitive: false
Editable By: ANY
Overrides:
Input 3:
Name: linkConfig.username
Label: Username
Help: Username that will be used to authenticate connection to SFTP serer.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 256
Input 4:
Name: linkConfig.password
Label: Password
Help: Password that will be used to authenticate connection to the FTP Server.
Type: STRING
Sensitive: true
Editable By: ANY
Overrides:
Size: 256
TO Job config 1:
Name: toJobConfig
Label: Output configuration
Help: Parameters required to store data on the SFTP server.
Input 1:
Name: toJobConfig.outputDirectory
Label: Output directory
Help: Directory on the SFTP server to write data to.
Type: STRING
Sensitive: false
Editable By: ANY
Overrides:
Size: 260
sqoop:000>
sqoop:000> create link -connector hdfs-connector 0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Creating link for connector with name hdfs-connector Please fill following values to create new link object Name: hdfs HDFS cluster URI: hdfs://127.0.0.1:9000 Conf directory: Additional configs:: There are currently 0 values in the map: entry# New link was successfully created with validation status OK and name hdfs sqoop:000>
sqoop:000> show link +------+----------------+---------+ | Name | Connector Name | Enabled | +------+----------------+---------+ | hdfs | hdfs-connector | true | +------+----------------+---------+
sqoop:000> create link -connector generic-jdbc-connector Creating link for connector with name generic-jdbc-connector Please fill following values to create new link object Name: mysql Database connection Driver class: com.mysql.jdbc.Driver Connection String: jdbc:mysql://127.0.0.1:3306/test Username: test Password: **** Fetch Size: Connection Properties: There are currently 0 values in the map: entry# SQL Dialect Identifier enclose: New link was successfully created with validation status OK and name mysql
sqoop:000> show link +-------+------------------------+---------+ | Name | Connector Name | Enabled | +-------+------------------------+---------+ | mysql | generic-jdbc-connector | true | | hdfs | hdfs-connector | true | +-------+------------------------+---------+
sqoop:000> create job -f "mysql" -t "hdfs" Creating job for links with from name mysql and to name hdfs Please fill following values to create new job object Name: from-mysql-to-hdfs Database source Schema name: test Table name: member SQL statement: Column names: There are currently 0 values in the list: element# Partition column: Partition column nullable: Boundary query: Incremental read Check column: Last value: Target configuration Override null value: Null value: File format: 0 : TEXT_FILE 1 : SEQUENCE_FILE 2 : PARQUET_FILE Choose: 0 Compression codec: 0 : NONE 1 : DEFAULT 2 : DEFLATE 3 : GZIP 4 : BZIP2 5 : LZO 6 : LZ4 7 : SNAPPY 8 : CUSTOM Choose: 0 Custom codec: Output directory: /sqoop/member Append mode: Throttling resources Extractors: Loaders: Classpath configuration Extra mapper jars: There are currently 0 values in the list: element# New job was successfully created with validation status OK and name from-mysql-to-hdfs
sqoop:000> show job +----+--------------------+--------------------------------+-----------------------+---------+ | Id | Name | From Connector | To Connector | Enabled | +----+--------------------+--------------------------------+-----------------------+---------+ | 1 | from-mysql-to-hdfs | mysql (generic-jdbc-connector) | hdfs (hdfs-connector) | true | +----+--------------------+--------------------------------+-----------------------+---------+
sqoop:000> start job -n from-mysql-to-hdfs sqoop:000> start job -n from-mysql-to-hdfs Submission details Job Name: from-mysql-to-hdfs Server URL: http://localhost:12000/sqoop/ Created by: hadoop Creation date: 2017-07-22 23:18:02 CST Lastly updated by: hadoop External ID: job_1499236611045_0001 http://iZj6ciilv2rcpgauqg2uuwZ:8088/proxy/application_1499236611045_0001/ 2017-07-22 23:18:02 CST: BOOTING - Progress is not available
啟動後進入HDFS查看導入情況
[hadoop@netkiller ~]$ hdfs dfs -ls /sqoop [hadoop@netkiller ~]$ hdfs dfs -ls /member Found 10 items -rw-r--r-- 3 hadoop supergroup 0 2017-07-22 23:18 /member/310af608-5533-4bc2-bfb8-eaa45470b04d.txt -rw-r--r-- 3 hadoop supergroup 48 2017-07-22 23:18 /member/36bc39a5-bc73-4065-a361-ff2d61c4922c.txt -rw-r--r-- 3 hadoop supergroup 0 2017-07-22 23:18 /member/3e855400-84a9-422d-b50c-1baa9666a719.txt -rw-r--r-- 3 hadoop supergroup 140 2017-07-22 23:18 /member/3e8dad92-e0f1-4a74-a337-642cf4e6d634.txt -rw-r--r-- 3 hadoop supergroup 55 2017-07-22 23:18 /member/4a9f47f1-0413-4149-a93a-ed8b51efbc87.txt -rw-r--r-- 3 hadoop supergroup 0 2017-07-22 23:18 /member/4dc5bfe7-1cd9-4d9b-96a8-07e82ed79a71.txt -rw-r--r-- 3 hadoop supergroup 0 2017-07-22 23:18 /member/60dbcc60-61f2-4433-af39-1dfdfc048940.txt -rw-r--r-- 3 hadoop supergroup 0 2017-07-22 23:18 /member/6d02ed89-94d9-4d4b-87ed-d5da9d2bf9fe.txt -rw-r--r-- 3 hadoop supergroup 209 2017-07-22 23:18 /member/cf7b7185-3ab6-4077-943a-26228b769c57.txt -rw-r--r-- 3 hadoop supergroup 0 2017-07-22 23:18 /member/f2e0780d-ad33-4b35-a1c7-b3fbc23e303d.txt
sqoop:000> update link -n mysql Updating link with name mysql Please update link: Name: mysql Database connection Driver class: com.mysql.jdbc.Driver Connection String: jdbc:mysql://127.0.0.1:3306/test Username: test Password: **** Fetch Size: Connection Properties: There are currently 0 values in the map: entry# SQL Dialect Identifier enclose: link was successfully updated with status OK