Big Data

Auto Added by WPeMatico

✅

A Coding Implementation on Spatial Graph Neural Networks for Urban Function Inference Using city2graph, OSMnx, and PyTorch Geometric

ai, AI (Artificial Intelligence), Artificial Intelligence, Big Data, Dataset, Editors Pick, Machine Learning, Staff, Technology, Tutorials

In this tutorial, we build an end-to-end spatial graph learning pipeline using city2graph. We start by collecting real urban POI data and street network information from OpenStreetMap, with a synthetic fallback to ensure the workflow remains reliable. We then engineer spatial features, construct multiple proximity graph families, and compare how different graph-building strategies represent the […]

A Coding Implementation on Spatial Graph Neural Networks for Urban Function Inference Using city2graph, OSMnx, and PyTorch Geometric Read More »

✅

Building a Code Dataset Pipeline from NVIDIA Nemotron-Pretraining-Code-v3 Metadata with Streaming, Pandas, and tiktoken

ai, AI (Artificial Intelligence), Artificial Intelligence, Big Data, Data Science, Editors Pick, Staff, Technology, Tutorials

In this tutorial, we work with NVIDIA’s Nemotron-Pretraining-Code-v3 dataset as a large-scale metadata index for code pretraining research. Instead of downloading the full multi-gigabyte dataset, we stream it, inspect its schema, and build a manageable sample for analysis. We then explore the dataset by studying languages, file extensions, repository frequency, and directory depth, which helps

Building a Code Dataset Pipeline from NVIDIA Nemotron-Pretraining-Code-v3 Metadata with Streaming, Pandas, and tiktoken Read More »

✅

A Coding Guide to Implement a pgvector-Powered Semantic, Hybrid, Sparse, and Quantized Vector Search System

ai, AI (Artificial Intelligence), Applications, Artificial Intelligence, Big Data, Data Science, Editors Pick, Staff, Technology, Tutorials

In this tutorial, we build a complete pgvector playground inside Google Colab and explore how PostgreSQL can work as a powerful vector database for modern AI applications. We start by installing PostgreSQL, compiling the pgvector extension, connecting through Psycopg, and registering vector types for smooth Python integration. Then, we create embeddings with SentenceTransformers, store them

A Coding Guide to Implement a pgvector-Powered Semantic, Hybrid, Sparse, and Quantized Vector Search System Read More »

PySpark Optimization: 12 Proven Techniques to Speed Up Your Spark Jobs

ai, AI (Artificial Intelligence), Artificial Intelligence, Beginner, Big Data, Database, Spark

Modern data pipelines handle massive volumes of structured and unstructured data every day. As datasets grow, poorly optimized Spark jobs become slower, more expensive, and harder to scale. Common issues include long execution times, excessive shuffling, memory bottlenecks, and inefficient joins. Effective PySpark optimization can significantly improve performance, reduce infrastructure costs, and enhance cluster efficiency.

PySpark Optimization: 12 Proven Techniques to Speed Up Your Spark Jobs Read More »

✅

How to Build Knowledge Graph Generation Pipelines From Text With kg-gen, NetworkX Analytics, and Interactive Visualizations

agentic ai, ai, AI (Artificial Intelligence), Artificial Intelligence, Big Data, Data Science, Editors Pick, Knowledge Graphs, Software engineering, Staff, Technology, Tutorials

In this tutorial, we will generate knowledge graphs from plain text, conversations, and multiple source documents using kg-gen. We start by setting up the required dependencies and configuring an LLM through LiteLLM, then we extract entities, predicates, and relationships from simple text. As we move forward, we work with longer passages using chunking and clustering,

How to Build Knowledge Graph Generation Pipelines From Text With kg-gen, NetworkX Analytics, and Interactive Visualizations Read More »

How to Build Technical Analysis and Backtesting Workflow with pandas-ta-classic, Strategy Signals, and Performance Metrics

ai, AI (Artificial Intelligence), Artificial Intelligence, Big Data, Data Science, Editors Pick, Staff, Technology, Tutorials

In this tutorial, we implement how to use pandas-ta-classic to build a complete technical analysis and trading strategy workflow. We start by installing the required libraries, downloading historical OHLCV stock data with yfinance, cleaning the returned data structure, and inspecting the available indicator categories inside the library. We then calculate popular indicators such as SMA,

How to Build Technical Analysis and Backtesting Workflow with pandas-ta-classic, Strategy Signals, and Performance Metrics Read More »

✅

How to Build a Single-Cell RNA-seq Analysis Pipeline with Scanpy for PBMC Clustering, Annotation, and Trajectory Discovery

ai, AI (Artificial Intelligence), Applications, Artificial Intelligence, Big Data, Data Science, Editors Pick, Software engineering, Staff, Technology, Tutorials

In this tutorial, we perform an advanced single-cell RNA-seq analysis workflow using Scanpy on the PBMC-3k benchmark dataset. We start by loading the dataset, inspecting its structure, and applying quality control checks to evaluate gene counts, total counts, mitochondrial content, and ribosomal gene signals. We then filter low-quality cells and genes, detect potential doublets with

How to Build a Single-Cell RNA-seq Analysis Pipeline with Scanpy for PBMC Clustering, Annotation, and Trajectory Discovery Read More »

Meta FAIR Releases NeuralSet: A Python Package for Neuro-AI That Supports fMRI, M/EEG, Spikes, and HuggingFace Embeddings

ai, AI (Artificial Intelligence), Artificial Intelligence, Big Data, Data Science, Editors Pick, Machine Learning, New Releases, Open Source, Software engineering, Staff, Tech News, Technology

Researchers at Meta’s FAIR lab have released NeuralSet, a Python framework designed to eliminate one of the most persistent bottlenecks in Neuro-AI research: the painful, fragmented process of getting brain data into a deep learning pipeline. https://kingjr.github.io/files/neuralset.pdf The Problem: Neuroscience Data Is Stuck in the Pre-Deep-Learning Era Neuroscience already has excellent, battle-tested software. Tools like

Meta FAIR Releases NeuralSet: A Python Package for Neuro-AI That Supports fMRI, M/EEG, Spikes, and HuggingFace Embeddings Read More »

A Coding Implementation on Document Parsing Benchmarking with LlamaIndex ParseBench Using Python, Hugging Face, and Evaluation Metrics

ai, AI (Artificial Intelligence), Artificial Intelligence, Big Data, Data Science, Editors Pick, Software engineering, Staff, Technology, Tutorials

In this tutorial, we explore how to use the ParseBench dataset to evaluate document parsing systems in a structured, practical way. We begin by loading the dataset directly from Hugging Face, inspecting its multiple dimensions, such as text, tables, charts, and layout, and transforming it into a unified dataframe for deeper analysis. As we progress,

A Coding Implementation on Document Parsing Benchmarking with LlamaIndex ParseBench Using Python, Hugging Face, and Evaluation Metrics Read More »

The LoRA Assumption That Breaks in Production 

ai, AI (Artificial Intelligence), Artificial Intelligence, Big Data, Data Science, Editors Pick, Technology

LoRA is widely used for fine-tuning large models because it’s efficient, but it quietly assumes that all updates to a model are similar. In reality, they’re not. When you fine-tune for style (like tone, format, or persona), the changes are simple and concentrated in just a few dimensions — which LoRA handles well with low-rank

The LoRA Assumption That Breaks in Production  Read More »