LLM Exploration – Explore HuggingFace Models and Others & Open-Source Model Integration

[peachjelly13]

LLM Exploration – Explore HuggingFace Models and Others & Open-Source Model Integration

Explore the integration of HuggingFace models and other open-source language models (LLMs) into our system. Assess the feasibility and scope of using these models for our needs, and document the findings.

Things to do

• Investigate the capabilities and features of HuggingFace models and other notable open-source LLMs.
• Assess the integration scope for incorporating these models into our system.
• Identify benefits and challenges associated with using these models (e.g., performance, scalability, accuracy, resource requirements).
• Explore any available pre-built libraries, SDKs, or tools for easier integration with these models.
• Document the integration process, potential use cases, and recommendations for leveraging LLMs in our project

End Goal: A comprehensive documentation file that outlines the integration scope, challenges, and a recommended approach for utilizing HuggingFace or other LLMs.

[HarshitBilagi]

I would like to take this on!
If anyone else is also interested, feel free to join me.

[peachjelly13]

Sure

[HarshitBilagi]

Here's the initial documentation that outlines the integration scope, challenges, and a recommended approach for integration of HuggingFace or other LLMs.
Documentation
Let me know whether or not the above work meets the goal of the issue.

[peachjelly13]

@HarshitBilagi Can you put this document on the group ? And open a discussion there

[HarshitBilagi]

Sure.

[satyajitnayk]

I went through the doc on Integrating LLMs for Enhanced Code Review—here are some of my thoughts:

1. User feedback
   Right now, the main way users can give feedback on AI-generated reviews would be through thumbs up👍/down👎. That’s a decent starting point, but over time we might want to capture more signals—like whether the suggestions are actually followed or edited.

2. Managing our own LLM
   Running our own model feels like a big lift, especially early on. It could lead to issues like lower accuracy or repetitive comments, which isn’t ideal since our focus is on correctness and quality. Hosted models seem more reliable for now.

3. Using the right model for the job
   For code reviews, we could use something like GPT-4o, which handles reasoning and context really well. And for security-specific feedback, maybe we plug in a more specialized model trained on secure coding practices.

4. Start modular
   If we design things in a modular way, we can start with hosted APIs and later switch to self-hosted or hybrid setups if it makes sense.

5. Avoiding comment spam
   We'll also need to be careful not to flood PRs with redundant or low-value comments—some sort of filtering or deduplication would be good to plan for early.

Feel free to correct me if I'm not in the right direction—open to thoughts! 🙂

[harsha-iiiv]

I agree with @satyajitnayk on few points here, let user use their own API keys of available LLMs. On point 5, we are still yet reach that state, but it is indeed true.

[HarshitBilagi]

These are excellent points @satyajitnayk , and I'm totally aligned with your thinking. The ideas around capturing more nuanced user feedback and strategically using different models make a lot of sense. Thanks for breaking this down! Please let me know how can we proceed based on current research!

[peachjelly13]

@HarshitBilagi we can create issues around these, but I feel there are some other things we would need to do before we reach till here

[HarshitBilagi]

Sure @peachjelly13, we'll proceed further once those things are done.