Apr 6Getting Started with Apache Beam: A Practical Example for Data ProcessingApache Beam is an open-source data processing framework that provides a unified programming model for both batch and stream processing. It was initially developed by Google and was later contributed to the Apache Software Foundation. …Data Engineering5 min readData Engineering5 min read
Mar 25What is Lambda Architecture Basics — Data EngineeringLambda architecture is a data processing framework that aims to provide a unified and fault-tolerant approach to big data processing. The architecture is designed to handle both batch and real-time data processing, providing a comprehensive solution for handling large-scale data analysis. …Data Engineering4 min readData Engineering4 min read
Feb 14What is the Difference between Clustered Index vs. Non-Clustered Index in SQLIndexes are important components in relational databases as they are used to improve the performance of database queries. In SQL, there are two main types of indexes: Clustered indexes Non-Clustered indexes Understanding the differences between these two types of indexes is crucial for database design and query optimization. Clustered Indexes: A clustered…Sql3 min readSql3 min read
Feb 12What is Row Level Security in SQL? What are Best Practices and How to Implement them?Row-level security (RLS) is a feature in SQL that allows you to restrict access to specific rows of a table based on a user’s role or permissions. RLS is used to enforce security and privacy policies in a database and prevent unauthorized access to sensitive data. To implement RLS, you…Sql4 min readSql4 min read
Jan 15How to implement MySQL Buffer Logic in a Scrapy Web CrawlerIn web crawlers, buffer logic refers to the set of rules or algorithms used to manage and manipulate the URLs that are being crawled. These rules determine how URLs are added to, removed from, and processed within the buffer. One example of buffer logic in web crawlers is using a…Buffer3 min readBuffer3 min read
Jan 12How to write Scrapy MySQL Data PipelineHere’s an example of how you might create a Scrapy pipeline that checks if a table exists in a MySQL database, and if it doesn’t, creates it: import mysql.connector from mysql.connector import errorcode from scrapy.exceptions import DropItem class MySqlPipeline: def __init__(self, host, user, password, database, table)…Web Scraping4 min readWeb Scraping4 min read
Mar 22, 202220 Best Data and Web Scraping ToolsData is the new oil. Nowadays, Everyone needs data whether you are running an e-commerce business, performing quantitative research, working in cyber threat intelligence, or blockchain, or else analyzing it and making better decisions. Data Scientists spend almost 50-80% of their time collecting and curating the data for the projects. …Web Scraping7 min readWeb Scraping7 min read
Jan 16, 2022How to Update/Partial Update data in Elasticsearch Using Python ClientTo update or partially update, a document in elasticsearch is a daunting task for the first time. There isn’t much of a clear tutorial or guide available out there. So, I thought to write one for the community. For pip install elasticsearch=7.16.3 Hire Me: Are you seeking a proficient individual with…Elasticsearch2 min readElasticsearch2 min read
Nov 30, 2021Approaching (Almost) Any Machine Learning Problem Book ReviewIn this book review, I will briefly discuss a few learnings from the book, why you should read, and how to approach it. First and foremost, the question is why you should read this book and put time and effort into it, In a world where there are tons of…Machine Learning4 min readMachine Learning4 min read
Nov 30, 2021How Emerging Technologies Interact with Each Other. Feat. Cloud, AI, Blockchain, IoT, Cyber Security, Quantum Computing, Federated Learning, Knowledge GraphsIt’s the year 2021, Above is the list of emerging technologies that are changing the face of the earth as we know it and will be very…Future Of Work1 min readFuture Of Work1 min read
