Search Results (15)

The arbitrary-order hidden Markov model ($\alpha$-HMM) is a nontrivial generalization of the standard HMM, designed to model stochastic processes with higher-order dependences among arbitrarily distant random events. The $\alpha$-HMM admits an efficient Viterbi-style optimal decoding algorithm, making it feasible to discover higher-order dependences among data objects in observed sequential data. Because the $\alpha$-HMM exceeds the expressive power of standard HMMs, fixed $k\mathrm{th}$-order HMMs, and stochastic context-free grammars, effective probabilistic parameter estimation approaches are required to translate this theoretical expressiveness of the $\alpha$-HMM into practical utility. This paper introduces a principled methodology for effective estimation of probabilistic parameters of the $\alpha$-HMM from observed data. In large-scale sequential datasets, higher-order dependencies can vary widely across instances, so a single global parameter set may be inadequate. Instead, an amortized parameter inference approach is proposed for the $\alpha$-HMM, in which an input-conditioned parameter estimator is learned from data and used to infer instance-specific parameters for each input instance to the decoding algorithm. Specifically, the neural parameter estimator is trained using a composite learning objective that is partially enabled by the optimal decoding algorithm. The effectiveness of the proposed parameter estimation method is demonstrated through empirical results of the application of the $\alpha$-HMM in biomolecular structure modeling and prediction. Full article

This study builds on a small body of work, all on Japanese, demonstrating how intonational phonology is critical for understanding prosodic modifications in infant-directed speech (IDS) relative to adult-directed speech. We performed similar analyses on simulated infant-directed speech vs. reading of a story in English and Bengali: two languages that – unlike Japanese – both have stress and do not use fundamental frequency (F0) to signal changes in word-level meaning, but that have two very different intonational grammars. These differences allowed us to disentangle previous hypotheses about intonational exaggeration in IDS being concentrated in a particular part of the melody. We tested hypotheses that state this locus of exaggeration is either at: the final position in the melody (final in the intonational phrase), the most unpredictable part of the melody, or in pragmatically informative tones. Our results support the first hypothesis. We found that the phonological choices of speakers to chunk the story into shorter, larger prosodic constituents drive intonational exaggeration in IDS. This is because the intonational phrase-final position in both languages is the site of greatest pre-boundary lengthening and F0 range expansion. We also demonstrate: (i) quantification of predictability in intonational melodies using probabilistic finite state automaton representations of intonational grammars and (ii) F0 statistical analyses that are robust and scalable to large, naturalistic IDS corpora. Full article

Maximum Entropy and Gradient Harmonic Grammar are well-established grammatical models for the analysis of linguistic phenomena, but we demonstrate that their probabilistic versions are inherently unidentifiable, employing parameters that cannot be uniquely determined from empirical data. Through examining Greek nominal stress patterns, we propose a reparameterization approach that introduces two identifiable, phonologically interpretable, parameters. We employ both point estimation and Bayesian inference to calculate their values. The latter approach yields reliable parameter estimates with quantified uncertainty that point estimation cannot offer. Our findings shed light on lesser-known aspects of Greek stress and offer a robust methodological framework for probabilistic phonological modeling across languages. Full article

As a crucial component of account protection system evaluation and intrusion detection, the advancement of password guessing technology encounters challenges due to its reliance on password data. In password guessing research, there is a conflict between the traditional models’ need for large training samples and the limitations on accessing password data imposed by privacy protection regulations. Consequently, security researchers often struggle with the issue of having a very limited password set from which to guess. This paper introduces a small-sample password guessing technique that enhances cross-domain features. It analyzes the password set using probabilistic context-free grammar (PCFG) to create a list of password structure probabilities and a dictionary of password fragment probabilities, which are then used to generate a password set structure vector. The method calculates the cosine similarity between the small-sample password set B from the target area and publicly leaked password sets $A_i$ using the structure vector, identifying the set $A_{max}$ with the highest similarity. This set is then utilized as a training set, where the features of the small-sample password set are enhanced by modifying the structure vectors of the training set. The enhanced training set is subsequently employed for PCFG password generation. The paper uses hit rate as the evaluation metric, and Experiment I reveals that the similarity between B and $A_i$ can be reliably measured when the size of B exceeds 150. Experiment II confirms the hypothesis that a higher similarity between $A_i$ and B leads to a greater hit rate of $A_i$ on the test set of B, with potential improvements of up to 32% compared to training with B alone. Experiment III demonstrates that after enhancing the features of $A_{max}$, the hit rate for the small-sample password set can increase by as much as 10.52% compared to previous results. This method offers a viable solution for small-sample password guessing without requiring prior knowledge. Full article

The rapid development of science and technology enables technological innovations to change the way of English oral learning. Based on the use of a large language model (LLM), we developed a novel dynamic evaluation system for oral English, LLM-DAELSL, which combines daily oral habits and a textbook outline. The model integrates commonly used vocabulary from everyday social speech and authoritative prior knowledge, such as oral language textbooks. It also combines traditional large-scale semantic models with probabilistic algorithms to serve as an oral assessment tool for undergraduate students majoring in English-related fields in universities. The model provides corrective feedback to effectively enhance the proficiency of English learners through guided training at any time and place. The technological principle of the model involves inputting prior template knowledge into the language model for reverse guidance and utilizing the textbooks provided by China’s Ministry of Education. The model facilitates the practice and evaluation of pronunciation, grammar, vocabulary, and fluency. The six-month tracking results showed that the oral proficiency of the system learners was significantly improved in the four aspects, which provides a reference for other language learning method developments. Full article

We present hidden abstract stack Markov models (HASMMs) with their learning process. The HASMMs we offer carry the more expressive nature of probabilistic context-free grammars (PCFGs) while allowing faster parameter fitting of hidden Markov models (HMMs). Both HMMs and PCFGs are widely utilized structured models, offering an effective formalism capable of describing diverse phenomena. PCFGs are better accommodated than HMMs such as for expressing natural language processing; however, HMMs outperform PCFGs for parameter fitting. We extend HMMs towards PCFGs for such applications, by associating each state of an HMM with an abstract stack, which can be thought of as the single-stack alphabet of pushdown automata (PDA). As a result, we leverage the expressive capabilities of PCFGs for such applications while mitigating the cubic complexity of parameter learning in the observation sequence length of PCFGs by adopting the bilinear complexity of HMMs. Full article

With the development of the Internet, information security has attracted more attention. Identity authentication based on password authentication is the first line of defense; however, the password-generation model is widely used in offline password attacks and password strength evaluation. In real attack scenarios, high-probability passwords are easy to enumerate; extremely low-probability passwords usually lack semantic structure and, so, are tough to crack by applying statistical laws in machine learning models, but these passwords with lower probability have a large search space and certain semantic information. Improving the low-probability password hit rate in this interval is of great significance for improving the efficiency of offline attacks. However, obtaining a low-probability password is difficult under the current password-generation model. To solve this problem, we propose a low-probability generator–probabilistic context-free grammar (LPG–PCFG) based on PCFG. LPG–PCFG directionally increases the probability of low-probability passwords in the models’ distribution, which is designed to obtain a degeneration distribution that is friendly for generating low-probability passwords. By using the control variable method to fine-tune the degeneration of LPG–PCFG, we obtained the optimal combination of degeneration parameters. Compared with the non-degeneration PCFG model, LPG–PCFG generates a larger number of hits. When generating ${10}^7$ and ${10}^8$ times, the number of hits to low-probability passwords increases by $50.4\%$ and $42.0\%$, respectively. Full article

Despite their well-known weaknesses, passwords are still the de-facto authentication method for most online systems. Due to its importance, password cracking has been vibrantly researched both for offensive and defensive purposes. Hashcat and John the Ripper are the most popular cracking tools, allowing users to crack millions of passwords in a short time. However, their rule-based cracking has an explicit limitation of depending on password-cracking experts to come up with creative rules. To overcome this limitation, a recent trend has been to apply machine learning techniques to research on password cracking. For instance, state-of-the-art password guessing studies such as PassGAN and rPassGAN adopted a Generative Adversarial Network (GAN) and used it to generate high-quality password guesses without knowledge of password structures. However, compared with the probabilistic context-free grammar (PCFG), rPassGAN shows inferior password cracking performance in some cases. It was also observed that each password cracker has its own cracking space that does not overlap with other models. This observation led us to realize that an optimized candidate dictionary can be made by combining the password candidates generated by multiple password generation models. In this paper, we suggest a deep learning-based approach called REDPACK that addresses the weakness of the cutting-edge cracking tools based on GAN. To this end, REDPACK combines multiple password candidate generator models in an effective way. Our approach uses the discriminator of rPassGAN as the password selector. Then, by collecting passwords selectively, our model achieves a more realistic password candidate dictionary. Also, REDPACK improves password cracking performance by incorporating both the generator and the discriminator of GAN. We evaluated our system on various datasets with password candidates composed of symbols, digits, upper and lowercase letters. The results clearly show that our approach outperforms all existing approaches, including rule-based Hashcat, GAN-based PassGAN, and probability-based PCFG. The proposed model was also able to reduce the number of password candidates by up to 65%, with only 20% cracking performance loss compared to the union set of passwords cracked by multiple-generation models. Full article

Increasing concerns about the prevalence of false information and fake news has led to calls for automated fact-checking systems that are capable of verifying the truthfulness of statements, especially on the internet. Most previous automated fact-checking systems have focused on the use of grammar rules only for determining the properties of the language used in statements. Here, we demonstrate a novel approach to the fact-checking of natural language text, which uses a combination of all the following techniques: knowledge extraction to establish a knowledge base, logical inference for fact-checking of claims not explicitly mentioned in the text through the verification of the consistency of a set of beliefs with established trusted knowledge, and a re-querying approach that enables continuous learning. The approach that is presented here addresses the limitations of existing automated fact-checking systems via this novel procedure. This procedure is as follows: the approach investigates the consistency of presented facts or claims while using probabilistic soft logic and a Knowledge Base, which is continuously updated through continuous learning strategies. We demonstrate this approach by focusing on the task of checking facts about family-tree relationships against a corpus of web resources concerned with the UK Royal Family. Full article

Text-based passwords are a fundamental and popular means of authentication. Password authentication can be simply implemented because it does not require any equipment, unlike biometric authentication, and it relies only on the users’ memory. This reliance on memory is a weakness of passwords, and people therefore usually use easy-to-remember passwords, such as “iloveyou1234”. However, these sample passwords are not difficult to crack. The default passwords of IoT also are text-based passwords and are easy to crack. This weakness enables free password cracking tools such as Hashcat and JtR to execute millions of cracking attempts per second. Finally, this weakness creates a security hole in networks by giving hackers access to an IoT device easily. Research has been conducted to better exploit weak passwords to improve password-cracking performance. The Markov model and probabilistic context-free-grammar (PCFG) are representative research results, and PassGAN, which uses generative adversarial networks (GANs), was recently introduced. These advanced password cracking techniques contribute to the development of better password strength checkers. We studied some methods of improving the performance of PassGAN, and developed two approaches for better password cracking: the first was changing the convolutional neural network (CNN)-based improved Wasserstein GAN (IWGAN) cost function to an RNN-based cost function; the second was employing the dual-discriminator GAN structure. In the password cracking performance experiments, our models showed 10–15% better performance than PassGAN. Through additional performance experiments with PCFG, we identified the cracking performance advantages of PassGAN and our models over PCFG. Finally, we prove that our models enhanced password strength estimation through a comparison with zxcvbn. Full article

Slow-hash algorithms are proposed to defend against traditional offline password recovery by making the hash function very slow to compute. In this paper, we study the problem of slow-hash recovery on a large scale. We attack the problem by proposing a novel concurrent model that guesses the target password hash by leveraging known passwords from a largest-ever password corpus. Previously proposed password-reused learning models are specifically designed for targeted online guessing for a single hash and thus cannot be efficiently parallelized for massive-scale offline recovery, which is demanded by modern hash-cracking tasks. In particular, because the size of a probabilistic context-free grammar (PCFG for short) model is non-trivial and keeping track of the next most probable password to guess across all global accounts is difficult, we choose clever data structures and only expand transformations as needed to make the attack computationally tractable. Our adoption of max-min heap, which globally ranks weak accounts for both expanding and guessing according to unified PCFGs and allows for concurrent global ranking, significantly increases the hashes can be recovered within limited time. For example, 59.1% accounts in one of our target password list can be found in our source corpus, allowing our solution to recover 20.1% accounts within one week at an average speed of 7200 non-identical passwords cracked per hour, compared to previous solutions such as oclHashcat (using default configuration), which cracks at an average speed of 28 and needs months to recover the same number of accounts with equal computing resources (thus are infeasible for a real-world attacker who would maximize the gain against the cracking cost). This implies an underestimated threat to slow-hash protected password dumps. Our method provides organizations with a better model of offline attackers and helps them better decide the hashing costs of slow-hash algorithms and detect potential vulnerable credentials before hackers do. Full article

We show that the mutual information between two symbols, as a function of the number of symbols between the two, decays exponentially in any probabilistic regular grammar, but can decay like a power law for a context-free grammar. This result about formal languages is closely related to a well-known result in classical statistical mechanics that there are no phase transitions in dimensions fewer than two. It is also related to the emergence of power law correlations in turbulence and cosmological inflation through recursive generative processes. We elucidate these physics connections and comment on potential applications of our results to machine learning tasks like training artificial recurrent neural networks. Along the way, we introduce a useful quantity, which we dub the rational mutual information, and discuss generalizations of our claims involving more complicated Bayesian networks. Full article

Spam reviews are increasingly appearing on the Internet to promote sales or defame competitors by misleading consumers with deceptive opinions. This paper proposes a co-training approach called CoSpa (Co-training for Spam review identification) to identify spam reviews by two views: one is the lexical terms derived from the textual content of the reviews and the other is the PCFG (Probabilistic Context-Free Grammars) rules derived from a deep syntax analysis of the reviews. Using SVM (Support Vector Machine) as the base classifier, we develop two strategies, CoSpa-C and CoSpa-U, embedded within the CoSpa approach. The CoSpa-C strategy selects unlabeled reviews classified with the largest confidence to augment the training dataset to retrain the classifier. The CoSpa-U strategy randomly selects unlabeled reviews with a uniform distribution of confidence. Experiments on the spam dataset and the deception dataset demonstrate that both the proposed CoSpa algorithms outperform the traditional SVM with lexical terms and PCFG rules in spam review identification. Moreover, the CoSpa-U strategy outperforms the CoSpa-C strategy when we use the absolute value of decision function of SVM as the confidence. Full article

Ray J. Solomonoff died on December 7, 2009, in Cambridge, Massachusetts, of complications of a stroke caused by an aneurism in his head. Ray was the first inventor of Algorithmic Information Theory which deals with the shortest effective description length of objects and is commonly designated by the term “Kolmogorov complexity.” In the 1950s Solomonoff was one of the first researchers to treat probabilistic grammars and the associated languages. He treated probabilistic Artificial Intelligence (AI) when “probabilistic” was unfashionable, and treated questions of machine learning early on. But his greatest contribution is the creation of Algorithmic Information Theory. [...] Full article

The surprisal of a word on a probabilistic grammar constitutes a promising complexity metric for human sentence comprehension difficulty. Using two different grammar types, surprisal is shown to have an effect on fixation durations and regression probabilities in a sample of German readers’ eye movements, the Potsdam Sentence Corpus. A linear mixed-effects model was used to quantify the effect of surprisal while taking into account unigram frequency and bigram frequency (transitional probability), word length, and empirically-derived word predictability; the so-called “early” and “late” measures of processing difficulty both showed an effect of surprisal. Surprisal is also shown to have a small but statistically non-significant effect on empirically-derived predictability itself. This work thus demonstrates the importance of including parsing costs as a predictor of comprehension difficulty in models of reading, and suggests that a simple identification of syntactic parsing costs with early measures and late measures with durations of post-syntactic events may be difficult to uphold. Full article