AWS Glue

Extract, transform, load (ETL) is the data integration process for loading information from one or more source databases into a data warehouse or a target database.rnIt consists of three functions or stages: • Extract: In this stage, data is read and extracted from the source database into a staging area. • Transform: Here, the raw data is checked, validated for any data integrity issues, and transformed so that it matches the data warehouse or target database schema. • Load: Finally, the transformed data is loaded into the data warehouse or target database.rnAWS Glue: Functionality and FeaturernThe main features of AWS Glue include:rnServerless computing:rnAWS Glue is a serverless offering. Serveless means users don’t have to manually designate a server to run it. Whenever the user wants to use AWS Glue service or functionality, Amazon spins up a server for its use and then shuts it down when it’s no longer in use. This automatic provisioning avoids scaling the infrastructure or the task of managing.rnApache Spark: AWS Glue is based on the Apache Spark analytics engine for big data processing. However, the service also allows users to create scripts in Scala and Python.rnEasy development: AWS Glue has access to “developer endpoints”: environments in which users can develop and test your AWS Glue scripts for the users who have decided to manually write their ETL code.rnAWS Glue Data Catalog: The AWS Glue Data Catalog is a metadata repository that stores information about sources and all of the user’s data stores giving the user more visibility into data assets regardless of location.rnJob scheduling: AWS Glue makes the task of scheduling easier by allowing the user to start jobs based on a schedule, an event, or completely on-demand.rnSome of the most recent AWS Glue updates are as follows: • From June 2019, Support for Python 3.6 in Python shell jobsrn• From May 2019, Support for connecting directly to AWS Glue via a virtual private cloud (VPC) endpoint. • From May 2019, Support for real-time, continuous logging for AWS Glue jobs with Apache Sparkrn• From March 2019, Support for custom CSV classifiers to infer the schema of CSV datarnWho uses AWS Glue?rnCompanies reportedly use AWS Glue in their tech stacks are as follow: • Tessianrn• iOLAPrn• Postmatesrn• Chimern• Depoprn• SparkPostrn• BizongornAWS Glue IntegrationsrnBelow is a list of tools that integrate with AWS Glue. • MySQLrn• Amazon S3rn• Amazon RDSrn• Microsoft SQLrn• Oraclern• Amazon Redshiftrn• Amazon RDSrn• Amazon EMRrnAWS Glue Alternativesrn Below are alternatives to Amazon Glue: • AWS Data Pipelinern• Airflowrn• Apache Sparkrn• Talendrn• AloomarnFeatures:rnFocus: Data catalog, ETLrnDatabase replication: Full table; incremental via change data capture through AWS Database Migration Service (DMS)rnSaaS sources: NonernThe ability for customers to add new data sources: Developers can write custom Python or Scala code and import custom libraries and Jar files into Glue ETL jobs to access data sources not natively supported by AWS Glue.rnConnects to data warehouses? Data lakes? Yes / YesrnTransparent pricing yesrnG2 customer satisfaction 4.1/5rnSupport SLAs AvailablernPurchase process Options for self-service and talking with salesrnCompliance, governance, and security certifications HIPAA, GDPRrnData sharing Yes, within AWSrnVendor lock-inAWS Glue is strongly tied to the AWS platform. Usage is billed monthly.