A study compared the vulnerability of C programs generated by nine state-of-the-art Large Language Models (LLMs) using a zero-shot prompt. The researchers introduced FormAI-v2, a dataset of 331,000 C programs generated by these LLMs, and found that at least 62.07% of the generated programs contained vulnerabilities, detected via formal verification. The research highlights the need for risk assessment and validation when deploying LLM-generated code in production environments.
Amir Goharshady from Hong Kong University of Science and Technology presented a talk at MBZUAI on martingale-based verification of probabilistic programs. The talk covered using martingale-based approaches for proving termination and synthesizing cost bounds for probabilistic programs, automating program analysis with template-based methods. He also discussed remaining challenges and open problems in the area. Why it matters: Advances in formal verification and analysis of probabilistic programs are crucial for ensuring the reliability and safety of AI systems that rely on randomization.
Dr. Youcheng Sun from the University of Manchester presented on ensuring the trustworthiness of AI systems using formal verification, software testing, and explainable AI. He discussed applying these techniques to challenges like copyright protection for AI models. Dr. Sun's research has been funded by organizations including Google, Ethereum Foundation, and the UK’s Defence Science and Technology Laboratory. Why it matters: As AI adoption grows in the GCC, ensuring the safety, dependability, and trustworthiness of these systems is crucial for public trust and responsible innovation.