Concerns around Intentional bias in GenAI by Data Poisoning | SNM Consulting

Data poisoning in AI tools refers to the malicious manipulation of training data used to train machine learning models. It involves introducing deliberately crafted or manipulated data into the training dataset with the intent to deceive or compromise the performance and reliability of the AI system. Before talking further about how data poisoning can happen and the far-reaching implications of the same, let’s recap what a machine learning model is and what the key attributes of the model A machine learning model is a mathematical algorithm that learns patterns and relationships in data to make predictions or decisions without being explicitly programmed. It takes input data, processes it through a series of mathematical operations, and produces an output based on the learned patterns and parameters.   In computer science, GIGO stands for “garbage in, garbage out.” It is a principle that highlights the fact that the quality of output from a computer system or algorithm is determined by the quality of the input data provided to it. The “garbage in, garbage out” (GIGO) principle applies to GenAI (generative artificial intelligence). If poor quality or biased data is used to train a GenAI model, it can produce flawed or biased outputs, perpetuating or amplifying existing biases, misinformation, or undesirable content. This can lead to negative consequences, such as the spread of false information, reinforcement of stereotypes, or the generation of inappropriate or offensive content. It underscores the importance of using high-quality, diverse, and carefully curated data for training GenAI models to mitigate these risks. In the article “What is Data Poisoning & Why Should You Be Concerned?” in the International Security Journal, the author “James Thorpe” says:

• The most significant downside to AI is that its efficacy is almost directly proportional to its data quality.
• Poor-quality information will produce subpar results, no matter how advanced the model is and history shows that it doesn’t take much to do this

Data poisoning attacks typically involve:

1. Identify the target model: Malicious actors typically choose a specific machine learning model or system that they intend to compromise. They study the target system and analyze its vulnerabilities and the data it relies on.
2. Craft poisoned data: The attackers generate or modify training data to inject misleading or malicious patterns. This data is carefully designed to exploit vulnerabilities in the learning algorithms used by the target system.
3. Inject the poisoned data: The attackers aim to introduce the poisoned data into the target system’s training dataset. This can be done by submitting fraudulent or manipulated data through various means, such as user interactions, automated scripts, or compromised sources.
4. Evade detection: To avoid detection during the training process, attackers may employ evasion techniques. These techniques are designed to make the poisoned data appear legitimate and blend in with the rest of the training dataset.
5. Impact the model’s behavior: By influencing the training process, the malicious actors aim to manipulate the model’s behavior or compromise its performance. The specific objectives can vary, such as causing misclassification, biased predictions, or targeted vulnerabilities.

Here are some “plausible” examples of data poisoning:

1. Image recognition: In 2017, researchers demonstrated a data poisoning attack on an image recognition system. By injecting subtle perturbations into a small percentage of training images, they were able to deceive the model into misclassifying those images during inference.
2. Spam filters: Spammers can employ data poisoning techniques by intentionally sending mislabeled emails to bypass spam filters. By carefully crafting email content and headers to resemble legitimate messages, spammers can manipulate the training data and evade detection.
3. Autonomous vehicles: Adversarial stickers or signs can be strategically placed on objects or roads to mislead autonomous vehicles. By subtly modifying stop signs or traffic signals, attackers can cause a self-driving car to misinterpret the scene and potentially lead to accidents.
4. Malware detection: Attackers can generate adversarial samples by modifying malware files to evade detection by antivirus software. By injecting noise or obfuscating certain parts of the file, they can trick the machine learning model used for malware classification into mislabeling the sample.
5. Online review systems: Competitors or malicious individuals may post fake positive or negative reviews to manipulate the ratings and rankings of products or services. These fake reviews can influence user decisions and deceive recommendation systems.
6. Voice assistants: By injecting subtle audio perturbations or background noise, attackers can manipulate voice commands to voice assistants. These perturbations can make the assistant misinterpret the command or perform unintended actions.
7. Financial fraud detection: Attackers can manipulate training data for fraud detection models by injecting synthetic or altered transactions to evade detection. This can include creating fake accounts or modifying transaction records to resemble legitimate activities.
8. Facial recognition systems: Data poisoning attacks can be performed by subtly modifying facial images used for training. By perturbing certain facial features, attackers can deceive the model into misclassifying or failing to recognize specific individuals.