Sunday, May 19, 2024
HomeBig DataSpeed up analytics on Amazon OpenSearch Service with AWS Glue by its...

Speed up analytics on Amazon OpenSearch Service with AWS Glue by its native connector

As the quantity and complexity of analytics workloads proceed to develop, prospects are searching for extra environment friendly and cost-effective methods to ingest and analyse knowledge. Knowledge is saved from on-line techniques such because the databases, CRMs, and advertising techniques to knowledge shops akin to knowledge lakes on Amazon Easy Storage Service (Amazon S3), knowledge warehouses in Amazon Redshift, and purpose-built shops akin to Amazon OpenSearch Service, Amazon Neptune, and Amazon Timestream.

OpenSearch Service is used for a number of functions, akin to observability, search analytics, consolidation, value financial savings, compliance, and integration. OpenSearch Service additionally has vector database capabilities that allow you to implement semantic search and Retrieval Augmented Technology (RAG) with giant language fashions (LLMs) to construct suggestion and media serps. Beforehand, to combine with OpenSearch Service, you can use open supply shoppers for particular programming languages akin to Java, Python, or JavaScript or use REST APIs supplied by OpenSearch Service.

Motion of information throughout knowledge lakes, knowledge warehouses, and purpose-built shops is achieved by extract, rework, and cargo (ETL) processes utilizing knowledge integration companies akin to AWS Glue. AWS Glue is a serverless knowledge integration service that makes it simple to find, put together, and mix knowledge for analytics, machine studying (ML), and utility improvement. AWS Glue gives each visible and code-based interfaces to make knowledge integration easy. Utilizing a local AWS Glue connector will increase agility, simplifies knowledge motion, and improves knowledge high quality.

On this put up, we discover the AWS Glue native connector to OpenSearch Service and uncover the way it eliminates the necessity to construct and keep customized code or third-party instruments to combine with OpenSearch Service. This accelerates analytics pipelines and search use instances, offering prompt entry to your knowledge in OpenSearch Service. Now you can use knowledge saved in OpenSearch Service indexes as a supply or goal inside the AWS Glue Studio no-code, drag-and-drop visible interface or straight in an AWS Glue ETL job script. When mixed with AWS Glue ETL capabilities, this new connector simplifies the creation of ETL pipelines, enabling ETL builders to save lots of time constructing and sustaining knowledge pipelines.

Resolution overview

The brand new native OpenSearch Service connector is a robust device that may assist organizations unlock the total potential of their knowledge. It lets you effectively learn and write knowledge from OpenSearch Service without having to put in or handle OpenSearch Service connector libraries.

On this put up, we show exporting the New York Metropolis Taxi and Limousine Fee (TLC) Journey Report Knowledge dataset into OpenSearch Service utilizing the AWS Glue native connector. The next diagram illustrates the answer structure.

By the tip of this put up, your visible ETL job will resemble the next screenshot.


To comply with together with this put up, you want a working OpenSearch Service area. For setup directions, seek advice from Getting began with Amazon OpenSearch Service. Guarantee it’s public, for simplicity, and observe the first consumer and password for later use.

Observe that as of this writing, the AWS Glue OpenSearch Service connector doesn’t assist Amazon OpenSearch Serverless, so it’s worthwhile to arrange a provisioned area.

Create an S3 bucket

We use an AWS CloudFormation template to create an S3 bucket to retailer the pattern knowledge. Full the next steps:

  1. Select Launch Stack.
  2. On the Specify stack particulars web page, enter a reputation for the stack.
  3. Select Subsequent.
  4. On the Configure stack choices web page, select Subsequent.
  5. On the Overview web page, choose I acknowledge that AWS CloudFormation may create IAM assets.
  6. Select Submit.

The stack takes about 2 minutes to deploy.

Create an index within the OpenSearch Service area

To create an index within the OpenSearch service area, full the next steps:

  1. On the OpenSearch Service console, select Domains within the navigation pane.
  2. Open the area you created as a prerequisite.
  3. Select the hyperlink beneath OpenSearch Dashboards URL.
  4. On the navigation menu, select Dev Instruments.
  5. Enter the next code to create the index:
PUT /yellow-taxi-index
  "mappings": {
    "properties": {
      "VendorID": {
        "sort": "integer"
      "tpep_pickup_datetime": {
        "sort": "date",
        "format": "epoch_millis"
      "tpep_dropoff_datetime": {
        "sort": "date",
        "format": "epoch_millis"
      "passenger_count": {
        "sort": "integer"
      "trip_distance": {
        "sort": "float"
      "RatecodeID": {
        "sort": "integer"
      "store_and_fwd_flag": {
        "sort": "key phrase"
      "PULocationID": {
        "sort": "integer"
      "DOLocationID": {
        "sort": "integer"
      "payment_type": {
        "sort": "integer"
      "fare_amount": {
        "sort": "float"
      "further": {
        "sort": "float"
      "mta_tax": {
        "sort": "float"
      "tip_amount": {
        "sort": "float"
      "tolls_amount": {
        "sort": "float"
      "improvement_surcharge": {
        "sort": "float"
      "total_amount": {
        "sort": "float"
      "congestion_surcharge": {
        "sort": "float"
      "airport_fee": {
        "sort": "integer"

Create a secret for OpenSearch Service credentials

On this put up, we use primary authentication and retailer our authentication credentials securely utilizing AWS Secrets and techniques Supervisor. Full the next steps to create a Secrets and techniques Supervisor secret:

  1. On the Secrets and techniques Supervisor console, select Secrets and techniques within the navigation pane.
  2. Select Retailer a brand new secret.
  3. For Secret sort, choose Different sort of secret.
  4. For Key/worth pairs, enter the consumer title opensearch.internet.http.auth.consumer and the password opensearch.internet.http.auth.go.
  5. Select Subsequent.
  6. Full the remaining steps to create your secret.

Create an IAM function for the AWS Glue job

Full the next steps to configure an AWS Id and Entry Administration (IAM) function for the AWS Glue job:

  1. On the IAM console, create a brand new function.
  2. Connect the AWS managed coverage GlueServiceRole.
  3. Connect the next coverage to the function. Substitute every ARN with the corresponding ARN of the OpenSearch Service area, Secrets and techniques Supervisor secret, and S3 bucket.
    "Model": "2012-10-17",
    "Assertion": [
            "Sid": "OpenSearchPolicy",
            "Effect": "Allow",
            "Action": [
            "Useful resource": [
            "Sid": "GetDescribeSecret",
            "Impact": "Enable",
            "Motion": [
            "Useful resource": "arn:aws:secretsmanager:<area>:<aws-account-id>:secret:<secret-name>"
            "Sid": "S3Policy",
            "Impact": "Enable",
            "Motion": [
            "Useful resource": [

Create an AWS Glue connection

Earlier than you should use the OpenSearch Service connector, it’s worthwhile to create an AWS Glue connection for connecting to OpenSearch Service. Full the next steps:

  1. On the AWS Glue console, select Connections within the navigation pane.
  2. Select Create connection.
  3. For Title, enter opensearch-connection.
  4. For Connection sort, select Amazon OpenSearch.
  5. For Area endpoint, enter the area endpoint of OpenSearch Service.
  6. For Port, enter HTTPS port 443.
  7. For Useful resource, enter yellow-taxi-index.

On this context, useful resource means the index of OpenSearch Service the place the information is learn from or written to.

  1. Choose Wan solely enabled.
  2. For AWS Secret, select the key you created earlier.
  3. Optionally, for those who’re connecting to an OpenSearch Service area in a VPC, specify a VPC, subnet, and safety group to run AWS Glue jobs contained in the VPC. For safety teams, a self-referencing inbound rule is required. For extra data, see Organising networking for improvement for AWS Glue.
  4. Select Create connection.

Create an ETL job utilizing AWS Glue Studio

Full the next steps to create your AWS Glue ETL job:

  1. On the AWS Glue console, select Visible ETL within the navigation pane.
  2. Select Create job and Visible ETL.
  3. On the AWS Glue Studio console, change the job title to opensearch-etl.
  4. Select Amazon S3 for the information supply and Amazon OpenSearch for the information goal.

Between the supply and goal, you may optionally insert rework nodes. On this resolution, we create a job that has solely supply and goal nodes for simplicity.

  1. Within the Knowledge supply properties part, specify the S3 bucket the place the pattern knowledge is positioned, and select Parquet as the information format.
  2. Within the Knowledge sink properties part, specify the connection you created within the earlier part (opensearch-connection).
  3. Select the Job particulars tab, and within the Primary properties part, specify the IAM function you created earlier.
  4. Select Save to save lots of your job, and select Run to run the job.
  5. Navigate to the Runs tab to verify the standing of the job. When it’s profitable, the run standing needs to be Succeeded.
  6. After the job runs efficiently, navigate to OpenSearch Dashboards, and log in to the dashboard.
  7. Select Dashboards Administration on the navigation menu.
  8. Select Index patterns, and select Create index sample.
  9. Enter yellow-taxi-index for Index sample title.
  10. Select tpep_pickup_datetime for Time.
  11. Select Create index sample. This index sample can be used to visualise the index.
  12. Select Uncover on the navigation menu, and select yellow-taxi-index.

You might have now created an index in OpenSearch Service and loaded knowledge into it from Amazon S3 in just some steps utilizing the AWS Glue OpenSearch Service native connector.

Clear up

To keep away from incurring prices, clear up the assets in your AWS account by finishing the next steps:

  1. On the AWS Glue console, select ETL jobs within the navigation pane.
  2. From the listing of jobs, choose the job opensearch-etl, and on the Actions menu, select Delete.
  3. On the AWS Glue console, select Knowledge connections within the navigation pane.
  4. Choose opensearch-connection from the listing of connectors, and on the Actions menu, select Delete.
  5. On the IAM console, select Roles within the navigation web page.
  6. Choose the function you created for the AWS Glue job and delete it.
  7. On the CloudFormation console, select Stacks within the navigation pane.
  8. Choose the stack you created for the S3 bucket and pattern knowledge and delete it.
  9. On the Secrets and techniques Supervisor console, select Secrets and techniques within the navigation pane.
  10. Choose the key you created, and on the Actions menu, select Delete.
  11. Cut back the ready interval to 7 days and schedule the deletion.


The mixing of AWS Glue with OpenSearch Service provides the highly effective capability to carry out knowledge transformation when integrating with OpenSearch Service for analytics use instances. This permits organizations to streamline knowledge integration and analytics with OpenSearch Service. The serverless nature of AWS Glue means no infrastructure administration, and also you pay just for the assets consumed whereas your jobs are working. As organizations more and more depend on knowledge for decision-making, this native Spark connector gives an environment friendly, cost-effective, and agile resolution to swiftly meet knowledge analytics wants.

In regards to the authors

Basheer Sheriff is a Senior Options Architect at AWS. He loves to assist prospects clear up attention-grabbing issues leveraging new expertise. He’s based mostly in Melbourne, Australia, and likes to play sports activities akin to soccer and cricket.

Shunsuke Goto is a Prototyping Engineer working at AWS. He works carefully with prospects to construct their prototypes and in addition helps prospects construct analytics techniques.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments