AI Chatbot: 46,000 Files with Tiered Access

Preface

This project details the development of an advanced chatbot designed to significantly improve customer support through automated, precise, and swift documentation retrieval. As a Full Stack AWS Developer, my role was to architect and implement a solution leveraging cutting-edge technologies such as Python, AWS serverless services (AWS Lambda, AWS Amplify, AWS S3, DynamoDB, Cognito), and a responsive front-end built with Next.js and Material-UI.

Tech Stack:

• Frontend: Next.js, Material-UI, JavaScript
• Backend: AWS Amplify, AWS Lambda, AWS Cognito, AWS S3, AWS DynamoDB
• Data Processing: Python, BeautifulSoup, Pinecone vector stores

Introduction

I developed a comprehensive large language model (LLM) chatbot solution for a UK-based ERP systems company. This chatbot enhanced customer experience by efficiently handling complex documentation queries and automating data management. My responsibilities included backend and frontend architecture, data scraping, and seamless integration with AWS cloud infrastructure.

Project Challenges

Scraping and Processing Large Document Repositories

A major challenge was efficiently scraping and managing data from a library of 46,000 documents. Python, combined with BeautifulSoup, was instrumental in extracting and structuring this vast dataset for optimal chatbot functionality.

Building Robust Multi-Tiered Authorisation

To support diverse user interactions—from basic queries to advanced file management—I designed a multi-tiered authorisation system. AWS Cognito managed authentication, while custom logic facilitated granular permission management.

Ensuring Cross-Device Responsive Design

Delivering a consistent user experience across mobile, tablet, and desktop required meticulous UI planning and execution. Next.js, coupled with Material-UI, enabled efficient creation of a responsive, intuitive frontend.

Rapid Skill Acquisition

The project required quickly mastering new technologies, notably Python and BeautifulSoup, to address the complex demands of document processing and data integration.

Technical Solutions Implemented

Python-Based Data Scraping

Using Python and BeautifulSoup, I successfully automated the scraping process for all 46,000 documents, significantly streamlining data preparation for the chatbot.

AWS Amplify Backend

AWS Amplify provided a scalable, secure infrastructure. Key components included:

• AWS Cognito: Managed secure user authentication and authorisation.
• AWS S3: Efficiently stored and managed large volumes of document data.
• AWS DynamoDB: Handled dynamic, scalable data management with high availability.

// Dynamic user navigation based on role\const getUserLinks = (userRole) => {
  const links = {
    MEMBER: fetchNavigation().memberLinks,
    PRIVILEGED_USER: fetchNavigation().privilegedLinks,
    ADMIN: fetchNavigation().adminLinks,
  };
  return links[userRole] || fetchNavigation().defaultLinks;
};

Responsive Frontend with Next.js and Material-UI

Next.js facilitated rapid development and server-side rendering, while Material-UI provided robust styling solutions to achieve responsive design.

// Responsive sidebar list item
import styled from 'styled-components';

const CustomListItem = styled.li.attrs(({ isActive }) => ({
  style: {
    backgroundColor: isActive ? 'blue' : 'transparent',
    color: isActive ? 'white' : 'black',
  },
}))`
  display: flex;
  flex-grow: 1;
  justify-content: center;
  padding: ${({ theme }) => theme.spacing(3)};
  cursor: pointer;
  border-radius: ${({ theme }) => theme.shape.borderRadius};
`;

export default CustomListItem;

Automated Vector Store Management with AWS Lambda

I implemented AWS Lambda functions triggered by AWS S3 events to automate updates to Pinecone vector stores. Scheduled overnight cron jobs ensured data consistency and reduced manual overhead, enhancing operational efficiency.

Project Outcomes

• Enhanced Efficiency: Reduced document retrieval times dramatically improved customer satisfaction.
• Scalable, Secure Infrastructure: AWS Amplify’s serverless architecture effortlessly managed scalability.
• Responsive and Engaging UI: A consistent user experience across devices significantly improved engagement metrics.
• Reliable Automation: Lambda-driven automation minimised manual maintenance, ensuring data integrity and reducing costs.

This project showcases my capability to architect and deliver sophisticated, scalable applications leveraging AWS services, modern JavaScript frameworks, and efficient data management solutions.

Project Images