Company Overview10Pearls is an end-to-end digital technology services partner helping businesses utilize technology as a competitive advantage. We help our customers digitalize their existing business, build innovative new products, and augment their existing teams with high performance team members. Our broad expertise in product management, user experience/design, cloud architecture, software development, data insights and intelligence, cyber security, emerging tech, and quality assurance ensures that we are delivering solutions that address business needs. 10Pearls is proud to have a diverse clientele including large enterprises, SMBs and high growth startups. We work with clients across industries, including healthcare/life sciences, education, energy, communications/media, financial services, and hi-tech. Our many long-term, successful partnerships are built upon trust, integrity and successful delivery and execution.RequirementsIdeal candidate should have a Bachelors degree in Computer Science with 3 5 years of programming in Snowflake, Aws, Glue, ETL pipeline Athena, Hadoop, Big DataResponsibilities
Develop, construct, test and maintain optimal data pipeline architecture
Assemble large, complex data sets that meet functional / non-functional business requirements.
Identify ways to improve data reliability, efficiency and quality
Prepare data for predictive and prescriptive modeling
Use data to discover tasks that can be automated
ETL pipeline build, deployment and operations, data quality, monitoring and alerting, CI/CD, governance, access control, etc
Design and implement efficient and scalable data pipelines, including data ingestion, transformation, and storage, using Snowflake and other relevant technologies.
Stay up-to-date with industry best practices and emerging technologies related to data architecture, data modeling, Snowflake, and AWS.
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Monitoring processes performance and advising any necessary changes
Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
Create data tools for analytics that assist in building and optimizing the product into an innovative industry leader
Work with data and analytics experts to strive for greater functionality in our data systems.
Demonstrate proficiency in data management and automation on Spark, Hadoop, and HDFS environments
Knowledge in DS/ML, analytics, or data warehousing
Required Skills
Bachelors or Masters in Computer Science, Data Engineering, or a related field.
3-5 years of experience in data engineering, with exposure to oil and gas data sources preferred.
Strong knowledge of SQL, Python, and data pipeline tools (e.g., Apache Airflow, Luigi).
Experience with cloud platforms (AWS, Azure, Google Cloud) and big data frameworks.
Familiarity with oil and gas-specific software and data types is a plus.