Maximilian Auch, Maximilian Balluff, Peter Mandl, and Christian Wolff, IAMLIS, Munich University of Applied Sciences HM, Lothstraße 34, 80335 Munich, Germany
We propose a novel method to classify GitHub repositories as professionally maintained or exploratory using their README.md files. We compare Large Language Models (LLMs) with classical NLP approaches like term frequency similarity and word embedding-based nearest neighbors, using RoBERTa as a baseline. We created and annotated a new dataset of over 200 repositories. Our evaluation shows LLMs outperform classical NLP models. GPT-4o achieved the best zero-shot classification without multi-step reasoning. Among smaller models, Google’s Gemini 1.5 Flash performed well. Few-shot learning improved performance for some models; Llama 3 (70b) reached 89.5% accuracy with multi-step reasoning, but improvements were inconsistent across models. Filtering based on word probability thresholds had mixed results. We discuss trade-offs between accuracy, time, and cost. Smaller models and prompt-based queries without multi-step reasoning offer faster, cost-effective solutions, useful in time-sensitive scenarios.Approximately 70% of repositories could be accurately classified based on README.md content.
Classification, README.md, Zero-shot, Few-shot, LLM.
Narcísio Mula1 and Claudio Nhancale2, 1Department of Mathematics, Universidade Save, Chongoene, Mozambique, 2 Department of Mathematics, Universidade Save, Chongoene, Mozambique
Cyber threats have rapidly evolved, rendering traditional security testing methods insufficient for the effective detection of vulnerabilities in software. This work proposes the development of an automated testing agent based on Machine Learning, aimed at enhancing the detection of vulnerabilities such as Cross-Site Scripting (XSS) and SQL Injection (SQLi). The study encompasses the collection and preparation of vulnerability data, as well as the selection and training of Machine Learning models, utilizing algorithms such as Support Vector Machines and Random Forests. Preliminary results indicate that the proposed approach improves accuracy in identifying vulnerabilities compared to traditional methods. This work contributes to the automation of security testing, providing a more adaptive and efficient solution to address the challenges of contemporary cyber threats.
Vulnerability Detection, Artificial Intelligence, Machine Learning .
