Unsafely coded browser extensions can compromise the security of a browser, making them attractive targets for attackers as a primary vehicle for conducting cyber-attacks. Among others, the three factors making vulnerable extensions a high-risk security threat for browsers include: i) the wide popularity of browser extensions, ii) the similarity of browser extensions with web applications, and iii) the high privilege of browser extension scripts. Furthermore, mechanisms that specifically target to mitigate browser extension-related attacks have received less attention as opposed to solutions that have been deployed for common web security problems (such as SQL injection, XSS, logic flaws, client-side vulnerabilities, drive-by-download, etc.). To address these challenges, recently some techniques have been proposed to defend extension-related attacks. These techniques mainly focus on information flow analysis to capture suspicious data flows, impose privilege restriction on API calls by malicious extensions, apply digital signatures to monitor process and memory level activities, and allow browser users to specify policies in order to restrict the operations of extensions. This article presents a model-based approach to detect vulnerable and malicious browser extensions by widening and complementing the existing techniques. We observe and utilize various common and distinguishing characteristics of benign, vulnerable, and malicious browser extensions. These characteristics are then used to build our detection models, which are based on the Hidden Markov Model constructs. The models are well trained using a set of features extracted from a number of browser extensions together with user supplied specifications. Along the course of this study, one of the main challenges we encountered was the lack of vulnerable and malicious extension samples. To address this issue, based on our previous knowledge on testing web applications and heuristics obtained from available vulnerable and malicious extensions, we have defined rules to generate training samples. The approach is implemented in a prototype tool and evaluated using a number of Mozilla Firefox extensions. Our evaluation indicated that the approach not only detects known vulnerable and malicious extensions, but also identifies previously undetected extensions with a negligible performance overhead.